User devices is a term that applies to computer systems such as desktop computers, smart televisions (TVs) and mobile computing devices such as laptop computers, mobile phones, tablets, “smart” watches, and glasses, to list a few examples. More specific examples of user devices include smartphones, tablet computing devices, and laptop computers running operating systems such as Windows, Android, Linux, or IOS, in examples.
Software that runs on the user devices can be generally classified as operating systems and software applications. Each user device allows the software applications to be installed on the user device, and then loads and executes the software applications upon the operating system of the user device, a process that is also known as deployment of the software applications to the user devices. The operating system of each user device manages the physical elements or hardware of the user devices such as its central processing unit (CPU) and peripherals. The operating system also presents well-defined Application Programming Interfaces (APIs) to the software applications. The software applications interact with the hardware of the user devices and can access its capabilities via the APIs of the operating system.
Software applications that interact with an operating system are also said to “run on top of” the operating system. Applications that run directly on top of an operating system are also known as native applications. One such software application is a virtual machine (VM).
VMs permit an isolated processing environment to exist on a user device. VMs present this isolated processing environment to other software applications that execute on top of the VMs. The VMs hide or abstract the details of the underlying hardware and/or operating system from the applications that run on top of the VMs. Applications that run on top of VMs of user devices are also known as “apps,” or user apps.
Host systems are computer systems that developers use to develop the user apps and prepare the user apps for execution on target user devices. Applications that run on top of VMs of host systems are also known as host applications, or host apps. Developers use host apps and native applications on the host systems to develop and compile the source code of the user apps. The host systems, as the name implies, host files for the user app on file systems of the host systems. Files of the user app include source files and compiled output files. The source files include the source code and resources of the user apps. The compiled output files include the compiled output of the source code, generated by a source code compiler. Often, the source code of the user apps is written using the Java programming language.
Java is a registered trademark of Oracle Corporation. Java source code is included within classes, and a Java compiler converts the source code for each Java class definition into its compiled output, also known as bytecode. Bytecode is typically a set of binary files that include platform-neutral instructions for implementing application behavior, where the bytecode is included in associated “class files” for each Java class definition.
Bytecode executes on VMs. The VMs interpret the bytecode, and execute corresponding native (e.g. machine-dependent) instructions on the target user device associated with the bytecode. Java user apps execute on user devices having Java-compatible VMs.
The term “platform” typically refers to an operating system of a user device or other computer system. One such platform is the Android platform. Android is a registered trademark of Google, Inc. Android is typically a mobile computing system operating system. It is based on the Linux kernel. User devices that support the Android platform/OS are also known as Android user devices. Other user device platforms include Apple IOS, Windows Phone, Raspberry Pi, and Linux, in examples. Regardless of the platform, the terms “classes” and/or “class files” universally refer to the bytecode created by that platform's toolchain for native execution of the user apps on the associated target user device.
While existing user device platforms enable deployment of user apps on user devices, the existing user device platforms typically cannot deploy the classes of the user apps directly. Instead, a software developer using a toolchain on a host system must first prepare the class files, resources, and other platform-specific files of the user app for inclusion within an archive. An archive is a file that has one or more computer files along with metadata. Archive files are used to collect multiple data files together into a single file for easier portability and storage, and/or to compress files to use less storage space. After the toolchain “packages” the class files of the user app into the archive, the toolchain then typically signs the archive using digital public key encryption. The signed archive is then sent to the user device, which decrypts the archive and unpacks its contents. Then, the user device can perform any final platform-specific processing of the unpacked class files prior to execution, a process known as class file optimization. Finally, a VM running on the user device loads the class files and starts the user app, thereby accomplishing the deployment of the user app to the user device.
Deployment of Windows Phone user apps, in one example, requires preparing either XAP or APPX archives including the classes and resources of the user app on a host system, and deploying the archives on a Windows user device having a Windows-compatible VM. The XAP archives are supported on user devices running Windows Phone 7/8/8.1/10 operating system, while the APPX archives are supported on user devices running Windows Phone 8.1 operating system or greater. Windows is a registered trademark of Microsoft, Inc.
In another example, developers of user apps for Apple user devices running the iOS operating system typically prepare archives known as bundles that include the class files and device specific resources of the user app. The Apple user app is then deployed by installing the bundle on an Apple user device, unpacking the class files and resources of the bundles and optimizing the unpacked class files on the Apple user device, and loading the class files on iOS-compatible VM running on the Apple user device.
Deployment of Android user apps, in yet another example, requires preparing an Android application package (APK) archive of the user app on a host system, and then deploying the user app to the user device via the APK of the user app. APK is an archive that is formatted in a fashion similar to ZIP and JAR archives that is used to distribute and install application software and middleware onto Android user devices. Preparation of an APK refers to the steps required to build and sign the APK on a host system to prepare the APK for deployment to a user device. This preparation includes converting the classes of the user app to be in an Android-specific format known as DEX files, including the DEX files along with other Android-specific resources and a manifest file within the APK, and digitally signing the APK. Deploying the user app typically requires installing the user app on the user device by sending the signed APK of the user app to the user device, decrypting the signed APK and unpacking its contents on the user device, optimizing the DEX file(s) of the APK on the user device, and then loading the optimized DEX files and/or resources of the user app into an Android-compatible VM running on the user device and executing the optimized DEX files to execute/start the user app on the user device.
User apps that extend the functionality of Android devices are developed primarily in the Java programming language. Dalvik and Android Runtime (ART) are examples of Android-specific VMs upon which Android user apps execute.
The Dalvik VM cannot execute Java bytecode directly. Dalvik requires that the Java bytecode is first converted into a further form of Android-specific set of native instructions known as Dalvik bytecode/Dalvik executable. Dalvik bytecode is stored in files having a “.dex” extension, also known as DEX files. A tool called “dx” is used to convert Java .class files into the DEX files. For this reason, Android user app developers often use terminology such as “Android user apps are compiled into DEX files” as a shorthand for this process.
ART (Android Runtime) is a more recent VM for Android that improves upon Dalvik. ART accepts DEX files to maintain backwards compatibility to Dalvik. In addition, a new Android development tool suite called the Java Android Compiler Kit (Jack) can directly produce DEX files from Java source code.
Initial startup of a user app refers to the combined steps of the preparation of the APK of the user app and the deployment of the APK. For an initial startup, none of the results from a possible previous build is available to the preparation step.
Incremental startup of a user app refers to the preparation and deployment of the APK of the user app, where all or parts of the output from a previous APK preparation are available.
Optimizing a DEX file on the user device is the process performed by the Android platform to create a pre-verified and optimized file for native execution on the operating system of the user device. On the Dalvik VM, this process is referred to as DexOpt, which, which is really just a back door into the VM. It performs an abbreviated VM initialization process, loads zero or more DEX files from the bootstrap class path, and then sets about verifying and optimizing whatever it can from the target DEX. On the ART runtime, the optimization process is referred to as Dex2Oat, which is a process taken to verify and compile the DEX file into a platform-specific native code block that is highly optimized for the user device.
Current systems and methods for deploying user apps to user devices have problems. First, user apps typically take a significant amount of time to initially install and execute, also known as the initial deployment time of the user app. Additionally, incremental updates to user apps, also known as redeployment of user apps, also take a significant amount of time. The time associated with preparing and redeploying incremental updates to a user app is also known as the turnaround time for a user app.
Current systems and methods for deploying user apps have significant initial deployment times. Current systems that perform initial deployment of Android user apps, in one example, have to first compile the source code of the user app (thus creating DEX files). The systems then process the Android-specific resources, build the APK file, sign the APK, and transfer the APK to the user device. On the user device, the systems unpack the APK, optimize the DEX files, and then startup the user app. For large user apps, all of these operations can be very time consuming, which correspondingly results in a tedious and long initial deployment time.
Current systems and methods for redeploying user apps also have significant turnaround times. Turnaround time includes the time associated with each change and retest of a user app, as part of a compile-uninstall-deploy-start cycle. Long turnaround time is a major issue for developing user apps on Android user devices, in one example.
For incremental updates to Android user apps, developers first make changes to the source code and/or resources of the user app and compile the changes on a host system. The changes to the classes and/or the resources are saved to a host file system of the host system. A new APK of the user app is then created that includes the changes to the classes and/or resources of the user app. The APK is then signed. Then, on the user device, the currently running instance of the user app is stopped, and its current archive/package file is uninstalled. The developer then deploys the new archive/APK of the user app on the user device, where optimization of the DEX files of the new APK must be performed before the user app can begin execution. This process is repeated for each compile-uninstall-deploy-start cycle.
Typically, development of a user app requires applying hundreds or even thousands of these cycles in an iterative fashion. Turnaround time generally increases proportionally with the size of the user app being developed. While the turnaround time can be especially long when initially developing a user app, additions or bug fixes to an already released user app can also have significant turnaround times.
A few projects to date have been created to address the problem of long turnaround time of user apps. One project is Buck Exopackage by Facebook, and another is Mirror by Jimulabs. Another project is the mobile-install Hazel build system by Google, and yet another is Instant Run by Google. Finally, JRebel for Android also implements a mechanism for faster incremental installs of APKs. JRebel is a registered trademark of ZeroTurnaround USA, Inc. However, each of these projects have limitations. The benefits and limitations of each of the projects are described herein below.
Buck Exopackage (hereinafter, “Exopackage”) improves the turnaround time in some cases, but not all. In particular, the incremental install implementation in Exopackage enables the sending of individual DEX files to the user device to an already running user app on the device, thereby only requiring the compile code phase, wherein the set of changed DEX files is produced. Hence, this approach skips the steps of creation of a new APK, signing of the APK, and transfer of the full APK to the user device. Then, on the device, Exopackage performs DEX optimizations only for the just received DEX files, thus re-using the unaffected optimized DEX files.
However, Exopackage has many limitations. In one example, its incremental install capability works only if the user makes changes to pure code, for example Java source code. In contrast, any change to an Android resource, which is very common, will cause this approach to be insufficient. This is because changes to resources require that a new full APK be created, signed and transferred to the device, before the possible DEX file optimization can begin. Exopackage also has an intrusive setup that requires modification of the original production source code of the user app to leverage its features. This pollutes the production code of the user app. Exopackage also does not support changes to resources and is limited to one software development build system for Android, the Buck build system. Finally, Exopackage requires extensive changes to build files.
Mirror by Jimulabs provides some support for hot-swapping of Android resources. However, Mirror works only on artificial test apps using test data where it allows to preview changes on the user device. While a limited set of changes can be applied to a test app quickly, the initial as well as the incremental deployment of apps is inherited from the standard Android tool chain. As a result, Mirror developers are often faced with long turnaround times.
A more recent approach is Bazel mobile-install by Google for Android user apps. Bazel is an implementation of incremental install. For this purpose, Bazel strips away as much as possible from the original APK, including Android resources and DEX files while performing the initial install. The resulting (much smaller) APK file is then installed onto the user device with a precompiled service component that is ready to receive the missing Android resources and DEX files. On the next development cycle, when the user has made some changes to the user app, the mobile-install will only need to send deltas to the already installed user app on the user device.
While being a lot faster than a standard Android build, Bazel has many limitations. In one example, the resulting user app that Bazel deploys on the user device is not equivalent to that of the same app installed by an initial install. Most importantly, resources that are installed by Hazel are only available within the user app code itself by direct reference from within the user app code. If the system attempts to reference any resource provided by the app, it will obtain the resources from the latest initial install.
Instant Run is an Android emulator created by Google. Instant Run attempts to address the problem of turnaround time in the sense that it provides a limited set of changes to be applied without having to reinstall the APK on the user device. However, in many cases either an initial install or an incremental install is required. Instant Run does not provide any beneficial improvements over a standard incremental install, thus it will be bottlenecked by the same slow steps of compiling, signing, transferring of the APK to the user device, and DEX file optimizations as standard build tools.
The JRebel tool for Android is another productivity tool that addresses the issue of turnaround time. JRebel is a registered trademark of Zeroturnaround OÜ. The JRebel for Android tool provides some improvements for both initial and incremental install of Android user apps. It does so by allowing for hot-swapping of a large set of changes onto the device without having to re-install the user app. In some cases, a fresh install is required, though. In contrast to Bazel's mobile-install feature, JRebel for Android guarantees that a correct APK is installed onto the device even for an incremental install. However, JRebel is limited by the fact that it has to perform a signing and transfer of a new stripped-down version of the user APK, required for the sake of APK correctness. JRebel is also limited in that that the optimization of DEX files can only begin on the user device after the process of signing and transferring of the new APK has been performed.